Digital video capabilities can be incorporated into a wide range of devices, including digital televisions, digital direct broadcast systems, wireless communication devices, portable digital assistants (PDAs), laptop computers, desktop computers, direct two-way communication devices (sometimes referred to as “walkie-talkies”), and radiotelephones such as cellular, satellite or terrestrial-based radiotelephones. These devices can provide significant improvements over conventional analog video systems in creating, modifying, transmitting, storing, and playing full motion video sequences.
A number of video processing and coding techniques have been established for communicating digital video sequences. The Moving Picture Experts Group (MPEG), for example, has developed a number of video coding standards including MPEG-1, MPEG-2 and MPEG-4. Other standards include the ITU H.263 standard and the ITU H.264 standard. Furthermore, a number of proprietary standards have also been developed by various companies, and new standards continue to emerge and evolve.
Many of these video standards make use of data compression. For example, many video coding techniques utilize graphics and video compression algorithms designed to compress data by exploiting temporal or inter-frame correlation, to provide inter-frame compression. The inter-frame compression techniques exploit data redundancy across frames by converting pixel-based representations of image frames to motion representations. In addition, the video compression techniques often use similarities within image frames, referred to as spatial or intra-frame correlation, allowing for intra-frame compression in which the motion representations within an image frame can be further compressed. The intra-frame compression is typically based upon conventional processes for compressing still images, such as discrete cosine transform (DCT) transformation.
Prior to performing video compression, however, a number of “front-end” image processing techniques are often performed on the image frames of a video sequence. For example, front-end image processing techniques are often used on images generated from an image sensor. Examples of such image processing techniques include demosaicing, lens rolloff correction, scaling, color correction, color conversion, and spatial filtering, to name a few. The processing may improve visual image quality attributes such as tone reproduction, color saturation, hue reproduction and sharpness.
For example, some image sensors used in video applications sample a scene using a color filter array (CFA) arranged on a surface of the respective sensors. A variety of CFA patterns may be used, and a digital signal processor (DSP) may be used to obtain three color values for each photosite. However, in order to obtain high quality color video images, image processing techniques may be required. These image processing techniques are refereed to herein as “front-end” image processing insofar as the techniques generally precede image compression or other video coding by a video coding unit. In other literature, however, these image processing techniques may be referred to as “post processing” steps since the processing occurs after images are captured by the image sensors.
Unfortunately, these image processing steps can present problems for real-time video encoding in real time video telephony (VT) applications. For example, in order to ensure real-time transmission of video sequences, all of the image processing needs to be performed very quickly. Accordingly, real-time image processing may require a very high bandwidth memory and large memory capacity in order to ensure that the extensive image processing, typically by several different image processing modules, can be executed. Unfortunately, memory bandwidth and memory space is limited for most video coding devices.